Papers by Johannes Baptist Mols
MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs (2025.findings-acl)
Copied to clipboard
Kaustubh Deshpande, Ved Sirdeshmukh, Johannes Baptist Mols, Lifeng Jin, Ed-Yeremai Hernandez-Cardona, Dean Lee, Jeremy Kritz, Willow E. Primack, Summer Yue, Chen Xing
| Challenge: | Existing evaluation frameworks for large language models have limited coverage for multi-turn conversations . multi-turned conversations require accurate instruction following, context allocation, and in-context reasoning at the same time. |
| Approach: | They propose a benchmark to evaluate large language models' ability to conduct multi-turn conversations with humans. |
| Outcome: | The proposed benchmarks achieve near perfect scores on existing benchmarks but only a 41.4% accuracy on the frontier models. |